{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "57700440",
   "metadata": {},
   "source": [
    "## Transferring Files To and From an S3 Bucket\n",
    "\n",
    "Retrieve files from an Amazon S3 bucket before executing a task, then upload files to an S3 bucket after the task's execution.\n",
    "\n",
    "The following example workflow downloads an image file from an S3 bucket, processes the file's contents, then uploads the file back to the S3 bucket.\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "Upload a color image file to an S3 bucket. Make note of the unique S3 file path to use in the workflow.\n",
    "\n",
    "In this example, the S3 file path is `s3://covalent-howto-tmp/remote_{unprocessed_filename}`, where `{unprocessed_filename}` is a variable containing the name of the file.\n",
    "\n",
    "### Procedure\n",
    "\n",
    "1. Define two Covalent `FileTransfer` objects and a Covalent `S3` strategy object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "052ed080",
   "metadata": {},
   "outputs": [],
   "source": [
    "import covalent as ct\n",
    "from typing import List, Tuple\n",
    "from pathlib import Path\n",
    "from skimage import io, color\n",
    "\n",
    "strategy = ct.fs_strategies.S3()\n",
    "\n",
    "unprocessed_filename = \"unprocessed_file.png\"\n",
    "processed_filename = \"processed_file.png\"\n",
    "\n",
    "unprocessed_filepath = str(Path(unprocessed_filename).resolve())\n",
    "processed_filepath = str(Path(processed_filename).resolve())\n",
    "\n",
    "\n",
    "s3_source_path = f\"s3://covalent-howto-tmp/remote_{unprocessed_filename}\"\n",
    "s3_dest_path = f\"s3://covalent-howto-tmp/remote_{processed_filename}\"\n",
    "\n",
    "ft_1 = ct.fs.FileTransfer(s3_source_path, unprocessed_filepath, strategy=strategy) # order defaults to BEFORE\n",
    "ft_2 = ct.fs.FileTransfer(processed_filepath, s3_dest_path, strategy=strategy, order=ct.fs.Order.AFTER)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a213bf5",
   "metadata": {},
   "source": [
    "2. Define an electron to:\n",
    "    1. Download the unprocessed file from S3\n",
    "    2. Perform some processing on the contents\n",
    "    3. Upload the processed file to S3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cff1491b",
   "metadata": {},
   "source": [
    "Access the file paths inside the electron as shown below using the \"files\" keyword argument. Covalent injects the source and destination file paths of the `FileTransfer` objects into the `files` argument. In this case, the `files` variable is a list of tuples of the form `(<source-path>, <destination-path>)`. The list looks something like this:\n",
    "\n",
    "```python\n",
    "[('/remote_unprocessed_file.png', '/path/to/current/dir/unprocessed_file.png'), ('/path/to/current/dir/processed_file.png', '/remote_processed_file.png')]\n",
    "```\n",
    "\n",
    "The S3 path is omitted from the remote path in the list; it is applied automatically by the `FileTransfer` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4519a791",
   "metadata": {},
   "outputs": [],
   "source": [
    "@ct.electron(files=[ft_1, ft_2]) # ft_1 is done before the electron is executed; ft_2 is done after.\n",
    "def to_grayscale(files: List[Tuple[str]] = None):\n",
    "\n",
    "    # Get the downloaded file's path\n",
    "    image_path = files[0][1] # destination file path of first file transfer, downloaded before executing this electron\n",
    "    \n",
    "    # Convert the image to grayscale\n",
    "    img = io.imread(image_path)[:, :, :3] # limiting image to 3 channels\n",
    "    gray_img = color.rgb2gray(img)\n",
    "\n",
    "    # Save the grayscale image to the upload file path\n",
    "    gray_image_path = files[1][0] # source filepath of second file transfer, to be uploaded\n",
    "    io.imsave(gray_image_path, gray_img, mode=\"L\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f05a5759",
   "metadata": {},
   "source": [
    "3. Create and dispatch a lattice to run the electron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1c2d3589",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "COMPLETED\n"
     ]
    }
   ],
   "source": [
    "@ct.lattice\n",
    "def process_s3_data():\n",
    "    return to_grayscale()\n",
    "\n",
    "dispatch_id = ct.dispatch(process_s3_data)()\n",
    "status = ct.get_result(dispatch_id, wait=True).status\n",
    "print(status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eea96bae",
   "metadata": {},
   "source": [
    "Notes:\n",
    "\n",
    "- This example illustrates a typical pattern in which files are downloaded from remote storage, are processed, and the results are uploaded to the same remote storage. Other scenarios can of course be implemented with the Covalent `FileTransfer`, `FileTransferStrategy`, and `@electron` components.\n",
    "- The example puts all the tasks (file download, processing, file upload) in one electron. For a real-world scenario of any complexity, a better practice would be to put each task in its own electron."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "244061a2",
   "metadata": {},
   "source": [
    "### See Also\n",
    "\n",
    "[Transferring Local Files During Workflows](./file_transfers_for_workflows_local.ipynb)\n",
    "\n",
    "[Transferring Files To and From a Remote Host](./file_transfers_to_from_remote.ipynb)\n",
    "\n",
    "[Transferring Files To and From Azure Blob Storage](./file_transfers_to_from_azure_blob.ipynb)\n",
    "\n",
    "[Transferring Files To and From Google Cloud Storage](./file_transfers_to_from_gcp_storage.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  },
  "vscode": {
   "interpreter": {
    "hash": "bc46502ba46b416dacb76460388e15e75b05e952c208e57c8294206d4336b33a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
